PATENT 
450100-04811 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
APPLICATION FOR LETTERS PATENT 



TITLE: IMAGING METHOD AND IMAGING APPARATUS 

INVENTOR: Junichi OGIKUBO 



William S. Frommer 
Registration No. 25,506 
FROMMER LAWRENCE & HAUG LLP 
745 Fifth Avenue 
New York, New York 10151 
Tel. (212) 588-0800 



1 

Ib^GING METEO) AND BEGINS APPARATUS 
BACKGEOJND OF THE INVEKTICM 

Field of tiie Invention 

The invention relates to an imaging method and an imaging 

apparatus . 
Related Art 

In generation of contents on audio and/or video for broadcasting, 
it has been often done to generate the contents altering motion speed 
of a subject partially for only a set period, in order to obtain 
enhanced effects the creator wishes to create. 

In such the generation of contents altering the motion speed of 
the subject, a higher frame rate relevant to a reference frame rate is 
illustratively set to generate the contents, and then the contents are 
reproduced at the reference frame rate so that slow motion contents can 
be produced. Alternatively, a lower frame rate relevant to a reference 
frame rate is set to generate the contents, and then the contents are 
reproduced at the reference frame rate so that quick motion contents 
can be produced. Further, adapting the frame rate to be set and the 
frame rate at the reproduction allows the motion speed of the subject 
to be freely altered. 

Thus, the creator creates the contents for broadcasting using not 
only the contents generated at the reference frame rate but also the 
contents of altered motion speed of the subject, in order to obtain the 
enhanced effects the creator wishes to create when the contents are 
reproduced at the reference frame rate. 



It has been proposed in Japanese Patent Publication No. Hll- 
177903 that the video camera allows tme axis to be coirpressed or 
extended in order to generate the contents of altered motion speed of 
the subject by altering the frame rate as described above. 

When the contents for broadcasting are generated using not only 
the contents generated at the reference frame rate but also the 
contents of altered motion speed of the subject as material, it is not 
easily determined at editing which of frame rates each of the contents 
has been generated if merely the material image is displayed. Further, 
when it perforins the speed change on the contents of altered motion 
speed of the subject, it is impossible to determine whether or not the 
speed change can be performed without any deterioration in iitege 
quality of the displayed image. If a frame rate is set (hereinafter, 
referred to as, ''set frame rate'') to 10 times the reference frame rate, 
the reproduced contents are displayed one-tenth in a motion of subject 
if the contents generated at the set frame rate are reproduced at the 
reference frame rate. Here, it is conceivable that since the set frame 
rate is 10 times the reference frame rate when the creator wants to 
display the contents one-fifth in a motion of subject, the speed change 
can be easily performed without any deterioration in image quality of 
the displayed image if performing a frame-skipping for each frame. It, 
however, is irrpossible to determine whether or not the speed change can 
be performed without any deterioration in image quality of the 
displayed image, easily based on merely the displayed image. 

It is an object of the present invention to present an imaging 
method and an imaging apparatus wherein determining which of frame 
rates the imaging has been performed is easily allowed. 
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Sf^&S^ OF THE INVE29nCW 

According to an aspect of the present invention, an imaging 
apparatus corrprises imaging device for reading a signal captured out of 
an image pickup device so as to generate an image signal based on 
captured image, and signal processor for generating image data based on 
the image signal. The apparatus also comprises controller for 
controlling operation of at least one of the imaging device and the 
signal processor to set a variable frame rate of the image data to a 
set frame rate, and generating associated information for indicating at 
least the set frame rate. The apparatus further comprises transmitter 
for combining the associated information with the image data to 
transmit the combined ones. 

According to another aspect of the present invention, an imaging 
method conprises the steps of generating image data based on a signal 
read out of an image pickup device, setting a variable frame rate of 
the image data to a set frame rate, generating associated information 
including frame rate information for indicating the set frame rate, and 
caribining the associated information with the image data to transmit 
the canbined ones. 

In the present invention, when reading signal captured out of an 
image pickup device so as to generate image signal based on captured 
image, and generating image data based on the image signal, the 
controller controls operation of at least one of the imaging device and 
the signal processor to set a variable frame rate of the image data to 
a set frame rate and generates associated information for indicating at 
least the set frame rate. Thus, the transmitter combines the associated 



information with the linage data to transmit the combineci ones. Based on 
the associated information, determining which of frame rates the 
iinaging has been performed is easily allowed, and thus, the contents of 
altered motion speed of the subject can be reproduced. 

Further, a variable frame rate of the image data is freely set to 
a set frame rate by means of altering a frequency of reading the signal 
captured out of the image pickup device, controlling the signal 
processor to perform frame-skipping, or altering a frequency of reading 
the signal captured out of the image pickup device and controlling the 
signal processor to add the image data on a frame basis. 

Alternatively, according to the invention, a sub-frame number is 
added to each of the frames of the set frame rate included within one 
frame period of reference frame rate. The sub-frame number is included 
in said associated information. This allows the frame-skipping to be 
easily performed when the reproduction speed is changed by performing 
the frame-skipping on frame basis. 

EVirther, according to the present invention, the signal processor 
samples analog audio signal to generate audio data and the controller 
controls a sampling frequency of the analog audio signal in the signal 
processor based on the set frame rate. Then, the transmitter combines 
the associated information with the image data and the audio data to 
transmit the combined ones. This allows the audio to be reproduced at a 
variable speed fitting the image. 

Additionally, the imaging apparatus further comprises a signal 
recording apparatus wherein the transmitter transmits a signal 
combining the associated information with the image data to the signal 
recording apparatus recording the signal thus conbined on recording 



medium. 

The concluding portion of this specification particularly points 
out and directly claims the subject matter of the present invention. 
However those skill in the art will best understand both the 
organization and method of operation of the invention, together with 
further advantages and objects thereof, by reading the remaining 
portions of the specification in view of the accorpanying drawing (s) 
wherein like reference characters refer to like elements. 

BRIEF DESCRIPTICN OF THE DRAWINGS 

Fig. 1 is a diagram for illustrating a configuration of contents- 
providing system; 

Fig. 2 is a block diagram showing a configuration of imaging 

apparatus; 

Fig. 3 is a flowchart showing an offer operation of sub-frame 
number; 

Fig. 4 is an illustration illustrating SDI format; 

Fig. 5 is an illustration illustrating SDTI format; 

Fig- 6 is an illustration illustrating SDTI-CP format; 

Fig. 7 is a block diagram showing a configuration of another 

imaging apparatus; 

Figs. 8A through 8E are diagrams showing relationship (part 1) 
between the image data and the associated information; 

Figs. 9A through 9E are diagrams showing relationship (part 2) 
between the image data and the associated information; 

Fig. 10 is a block diagram showing a configuration of signal- 
recording apparatus; 
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Figs. IIA through IIC are illustrations each showing GUI 
representation at an editing operation; and 

Figs. 12A through 12 J are illustrations showing a reproduction 

operation of the image. 

pp TOTTjan DESaOPTICN OF CTEEERRED EMBODIMEaJTS 

Referring to the drawings, the invention will now be descrUoed in 
detail with reference to an embodiment according to the invention. Fig. 
1 shows an entire configuration of contents-providing system for 
providing the contents such as the contents for audio and/or video. An 
imaging apparatus 10 generates image data having its frame rate 
variably set and combines associated information including frame rate 
information on the frame rate of the iinage data with this image data to 
supply a signal recording apparatus 20 or an editing apparatus 30 with 
the combined ones as material data Dim. When the imaging apparatus 10 
is provided with an audio input apparatus 19, it generates audio data 
to supply the signal recording apparatus 20 or the editing apparatus 30 
with the audio data and the associated information as material data Dnn. 

The signal recording apparatus 20 receives the material data Dim 
and records it on recording medium. It also reproduces the material 
data rmn recorded on the recording medium and sillies the editing 
apparatus 30 with the reproduced data. The imaging apparatus 10 and the 
signal recording apparatus 20 may be composed as a combination 
camera/recorder system, be combined together each other as so-called 
"dockable" typed camera/recorder system, and be separately provided 
with them being connected through a cable. 

The editing apparatus 30 receives the material data DTm from the 
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signal recording apparatus 20 or the imaging apparatios 10 and performs 
editing process on the material data Dim to produce iitage data and 
audio data the editor wishes to create. The editing apparatus 30 
combines associated information including frame rate information with 
the image data and the audio data thus produced to generate contents- 
data DC for broadcasting, distribution, and the like. 

Fig. 2 is a block diagram showing a configuration of the imaging 
apparatus 10. Imaging unit 12 receives incident light through imaging 
lens system 11 and an image pickup device such as charge coupled device 
(CCD) mounted within the imaging unit 12 captures subject image on its 
image surface. The image pickup device generates imaged charge of the 
subject image though photoelectric transfer characteristics thereof. 
The imaging unit 12 also reads out the imaged charge thus generated on 
the basis of driving signal CR from a timing generator 142, which will 
be described later, and generates imaged signal Sp having a frame rate 
that corresponds to the driving signal CR to supply camera processing 
circuit 131 in signal processor 13 with the imaged signal Sp. 

Based on timing signal CT received from the timing generator 142, 
the camera processing circuit 131 performs various signal processing at 
timings synchronized with the imaged signal Sp. Illustratively, such 
the signal processing represents noise-reduction processing for 
reducing noise components from the imaged signal Sp by means of 
correlated dual sartpling or the like, transformation processing from 
the imaged signal Sp thus noise-reduced to digital image data, clamp 
processing of the image data, processing on shading correction and 
pilling-up of deficiency in the image pickup device, y correction, 
contour enhancement processing, knee correction, and the like. Further, 
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the camera processing circuit 131 performs other various signal 
processing under the process conditions based on the operation control 
signal CS received from imaging control circuit 141 of controller 14. 
Thus, the camera processing circuit 131 generates image data DV thus 
obtained according to various signal processing and supplies 
transmitting unit 15 with the image data DV. 

The timing generator 142 of controller 14 generates driving 
signal CR corresponding to operation control signal CS from the imaging 
control circuit 141 and supplies the imaging unit 12 with the driving 
signal CR, thereby allowing a reading-out frequency of imaged charge in 
the imaging unit 12 to be altered. It, then, also controls a frame rate 
of the imaged signal Sp to be set to a set frame rate FRs based on an 
operation signal PSa from user interface unit 16. For example, frame 
frequency, 59.94Hz or 29.97Hz in the case of NTSC system or frame 
frequency, 50Hz or 25Hz in the case of PAL system is set to the frame 
frequency of reference frame rate FRr and an operation such that the 
set frame rate FRs is set to the one k times the reference frame rate 
FRr is carried out (k is positive value, not limited to an integer) . If 
so, a frame rate of the imaged signal Sp is controlled to be equal to 
the one k times the reference frame rate FRr. 

Alternatively, altering a period of reading-out pulse (sensor 
gate pulse) for transferring imaged charge accumulated on each pixel 
from the respective pixels in an image pickup device such as CCD to 
transfer unit or the like allows reading-out frequency of imaged charge 
to be altered, thereby altering the frame rate. 

It may be adapted for Common Data Rate (CDR) system in this case. 
When the CDR system is used, available frame rate may be altered, and a 
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frame rate of the signal output from the CCD may be constant, thus 
making processed rate in the camera processing circuit 131 or the like 
constant. 

This CDR systan has been generally disclosed in PCT i^lication 
No. PCT/JP03/0051 filed on January 22, 2003. 

The tljning generator 142 also generates timing signal CT 
synchronized with the driving signal CR and supplies the camera 
processing circuit 131 and audio signal processing circuit 132 with the 
tiitdng signal CT. The timing generator 142 further generates frame rate 
information CM-FRs indicating the set frame rate FRs that is frame rate 
of image data DV and supplies the transmitting unit 15 with the frame 
rate information CM-ERs. The timing generator 142 also generates sub- 
frame number BN. This sub-frame number BN is identification number for 
identifying each frame included in each of the frame periods of the 
reference frame rate FRr when the set frame rate FRs is set to be 
higher than the reference frame rate EBr. The timing generator 142 
supplies the transmitting unit 15 with this sub-frame number BN as 
frame identification information CM-BN. 

Fig. 3 is a flowchart showing an offer operation of sub-frame 
number in the timing generator 142. The timing generator 142 performs a 
frequency division on oscillated signal of a predetermined frequency, 
for example, to set the set frame rate FRs so that the frame periods of 
the reference frame rate FRr and the set frame rate FRs may be 
synchronized with each other, thereby generating a frame reference 
timing indicating a generation of the driving signal CR and a 
breakpoint between the periods of frames of reference frame rate FRr 
based on the frame period of the set frame rate FRs. 
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At step STl, the timing generator 142 determines whether or not 
the frame reference timing is detected. If the frame reference tiitdng 
is detected, the process goes to step ST2. If no frame reference timing 
is detected, the process goes back to the step STl. 

When the frame reference timing is detected at the step STl, the 
process goes to the step ST2, as described above, where the timing 
generator 142 initializes the sub-frame number BN, thus setting the 
sub-frame number BN to an initial value, for example, 0. The process 

then goes to step ST3. 

At the step ST3, the timing generator 142 determines whether or 
not the frame reference timing has been detected during a period of 
time starting from detecting the frame reference timing and ending 
before one frame of period of the set frame rate FRs has been passed. 
If no frame reference timing is detected, the process goes to step ST4 
where the timing generator 142 adds 1 to the sub-frame number BN and 
then updates the added sub-frame number BN. The process then goes back 
to step ST3. Thus, when no frame reference timing has been detected 
before one frame of period of the set frame rate FRs has been passed, 
the sub-frame number BN is allocated in sequence to the set frame rate 
FRs for each of the frames of periods thereof. 

Thereafter, when the timing generator 142 detects the frame 
reference timing before one frame of period of the set frame rate FRs 
has been passed, the process goes back to the step ST2 where the sub- 
frame nxjrtiber EN is initialized. 

Every frame of period of the reference frame rate FRr, the sub- 
frame number BN may be added to the frame image of the set frame rate 
FRs provided during a frame period of time. 
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Referring back to Fig. 2, the imaging control circuit 141 of the 
controller 14 is connected with the user interface unit 16. When, in 
the imaging apparatus 10, switching operation and alteration operation 
of the frame rate are performed, the user interface unit 16 generates 
operation signal PSa corresponding to these operations and supplies the 
imaging control circuit 141 with the operation control signal PS. When 
receiving the operation signal PSa from external ecjuipnent such as a 
remote controller, not shown, the user interface unit 16 also supplies 
the imaging control circuit 141 with the operation signal PSa. 

The imaging control circuit 141 generates operation control 
signal CS to allow the imaging apparatus 10 to be operated according to 
the operation signal PSa based on the operation signal PSa received 
frcm the user interface unit 16. It also supplies the camera processing 
circuit 131 and the timing generator 142 with the operation control 
signal CS. 

The audio signal processing circuit 132 receives analog audio 
signal Sin. The audio signal processing circuit 132 also performs 
sampling process on the analog audio signal Sin based on timing signal 
CT received from the timing generator 142 to generate digital audio 
data DA and supply the transmitting unit 15 with the digital audio data 
DA. 

The transmitting unit 15 generates associated information EM 
including the frame rate information m-FBs and the frame 
identification information CM-BN and combines the image data DV and the 
audio data DA with the associated information CM to generate material 
data DTm. The transmitting unit 15 supplies the signal recording 
apparatus 20 or the editing apparatus 30 with the material data Dim. 
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The associated data CM may include not only information on the set 
frame rate FRs and the sub-frame number BN but also information on 
imaged date and tiine, imaged condition, iiteged contents, and the like. 

It is conceivable that as a way to combine the image data UJ and 
the audio data DA with the associated information, the associated 
information CM is inserted into a data stream of image or a header of 
the data stream when the image data CfV and the audio data DA are 
compressed to generate the material data OThi as the data stream. 

For example, when a moving picture compression system in Moving 
Picture Experts Group (MEBG) standardized as ISO (InterTiational 
organization for Standardization) /lEC (International Electrotechnical 
Commission) 13818-2 is utilized, the associated information CM may be 
inserted into a region for extension or the like provided in a picture 
layer, thereby allowing the associated information CM to be inserted on 
a picture basis. Alternatively, when a synchronization multiplexed 
system in MPEG standardized as ISO/IEC 13818-2 is utilized, the 
associated information CM may be inserted into an optional field or the 
like provided in a header in Packetized Elementary Stream (PES) . 

Further, it is conceivable that when SDI format standardized as 
SMPTE (Society of Motion Picture and Television Engineers) 259M, 
Television-lO-Bit 4:2:2 Component and 4fsc Composite Digital Signals- 
Serial Digital Interface, in order to transfer non-compressed image and 
audio data, SDTI format standardized as SMPTE 305M, Television-Serial 
Data Transport' Interface (SDII), in order to transfer compressed image 
and audio data, or SDTI-CP format standardized as SMPTE 326M, 
Television-SDTI Content Package Format (SDTI-CP) that obtained by 
further defining SDTI format is utilized, the associated information CM 
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nay be inserted into a signal having each fomat as UMID data 
standardized as SMPTE 330M, Television-Unique Material Identifier 
(UMID) . In the present invention, a way to combine the image data DV 
and the audio data DA with the associated information is not limited to 
the above ones. Various ways to do so are conceivable. 

UMID is an identifier for unicjuely identifying material data such . 
as image data and audio data. UMID defines Basic UMID or Extended UMID 
that is identifier as being obtained by adding Signature Metadata to 
Basic UMID. Basic UMID includes a label for identifying digital data, 
information for indicating whether or not overwrite or editing is 
performed on the iteterial data, a number for distinguishing the 
material data from each other, and the like. Signature Metadata 
includes information on material-data-forming date and time, correction 
information (information on difference in time) concerning material- 
data-forming time, location information indicative of latitudes, 
longitudinal, and altitude, information on name of organization, and 
the like. Signature Metadata also includes a region for user code, to 
which the associated information CM may be stored, for exaitple. 

The Extended UMID thus obtained is inserted into each of the 
signals having various formats. When SDI format is utilized, the 
Extended UMID is inserted into an ancillary data region, as shown in 
Fig. 4. mien SDTI format is utilized, the Extended UMID is inserted 
into an ancillary data region except for a header data portion, as 
shown in Fig. 5. When SDTI-CP format is utilized, the data to be 
inserted is inserted into a payload region on an item basis. 
Specifically, as shown in Fig. 6, system item composed of information 
on image and audio, picture item composed of image data, audio item 
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composed of audio data, and auxiliary (AUX) item cotposed of other data 
are inserted in sequence. In the system item, regions for metadata sets 
such as Package Metadata set. Picture Metadata set, audio Metadata set, 
and Auxiliary Metadata set are provided to allow the metadata to be 
inserted thereinto. This permits the extended UMID to be inserted into 
these regions. 

The above iitiaging apparatus 10 alters read-out frequency of 
imaged charge in the iinaging unit 12 to generate the material data Ulm 
of a predetemdned set frame rate FRs, thereby allowing the set frame 
rate FRs to be successively altered. If, however, the set frame rate 
FRS is altered solely in a stepwise, frame-skipping allows the material 
data Dttn of a predetermined set frame rate FRs to be generated. This is, 
generating image data DVa having a constant frame rate higher than the 
set frame rate ERs and extracting iitage data of only the set frame rate 
FRS from the image data DVa allows the material data DTin of a 
predetermined set frame rate FRs to be generated. 

Fig. 7 shows a configuration of another iinaging apparatus 10a 
performing such a generation. In Fig. 7, like reference niimbers refer 
to like elements shown in Fig. 2 and thus, the detailed description 

thereof is omitted. 

Timing generator 182 in controller 18 generates driving signal 
CRa corresponding to the highest value of the set frame rate FRs that 
is set through user interface unit 16 and supplies the imaging unit 12 
with the driving signal CRa. Based on the driving signal CRa, the 
imaging unit 12 generates imaged. signal Spa having a fixed frame rate 
ER3 higher than a reference frame rate. The imaging unit 12 then 
supplies camera processing circuit 131 with the imaged signal Spa. When 



15 



the set frame rate ET^ may be altered up to n times (where n is 
positive number) the reference fraine rate FRr, the iitaging unit 12 
generates the imaged signal Spa having a frame rate n times the 
reference frame rate FRr, and supplies the camera processing circuit 
131 with the imaged signal Spa. In other words, the imaging unit 12 
generates the imaged signal Spa having a fixed frame rate without any 
influence of the set frame rate FRs that is set through the user 

interface unit 16. 

The timing generator 182 also generates timing signal CTa 
synchronized with the driving signal CRa and supplies the camera 
processing circuit 131 and audio signal processing circuit 132 in 
signal processor 17 and effective frame signal generation circuit 183 
in the controller 18 with the timing signal CTa. 

The camera processing circuit 131 generates image data DVa of 
fixed frame rate FRq based on the imaged signal Spa and supplies the 
effective data selection circuit 171 with the image data DVa. Ihe audio 
signal processing circuit 132 carries out a sampling based on the 
timing signal CTa of a fixed frequency to generate audio data DAa and 
supplies the effective data selection circuit 171 with the audio data 
DAa. 

Imaging control circuit 181 generates a set information signal CF 
indicating a set frame rate FRs based on operational signal PSa 
received from the user interface unit 16 and supplies the effective 
frame signal generation circuit 183 with the set information signal CF. 

The effective frame signal generation circuit 183 extracts data 
on a frame basis from the image data DVa based on a ratio of the fixed 
value of frame rate FRq of the image data DVa to a value of the set 
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frame rate FRs indicated in the set information signal CF and generates 
extraction control signal CC for generating iinage data DV of the set 
fran« rate FRs. The effective frame signal generation circuit 183 also 
synchronizes the extraction signal CC with the tiining signal CTa and 
supplies the effective data selection circuit 171 with the synchronized 
extraction control signal CC. When the frame rate ET^q of the iitage data 
DVa is n times the reference frame rate FRr and the set frame rate FBs 
is n/2 times the reference frame rate FBr, the effective frame signal 
generation circuit 183 generates the extraction control signal CC for 
controlling data extraction on a frame basis every other frame from the 
ixTBge data DVa and supplies the effective data selection circuit 171 
with the extraction control signal CC synchronized with the timing 
signal CTa. 

The effective frame signal generation circuit 183 further 
generates frair^ rate information CM-E^ indicating the set frame rate 
FRs based on the set information signal CF and supplies the 
transmitting unit 15 with the frame rate information EM-FRs. Since the 
effective frame signal generation circuit 183 may determine a number of 
frame during a period of frame with the reference frame rate FRr 
according to the extraction control signal CC, the effective frame 
signal generation circuit 183 sets sub-frarre number on a frame during a 
period of each frame with the reference frame rate FRr. The effective 
frame signal generation circuit 183 then supplies the transmitting unit 
15 with this sub-frame number m as the frame identification 

information CM-BN. 

The effective data selection circuit 171 extracts the image data 
DVa and the audio data DAa of the frame indicated by the extraction 
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control signal CC and supplies the transmitting unit 15 with them as 
the image data DV and the audio data DA. It is conceivable that the 
effective frame signal generation circuit 183 may supply the effective 
data selection circuit 171 with the frame rate information CM-FRs 
indicating the set frame rate FRs, and the effective data selection 
circuit 171 may perform frame-skipping on the audio data DAa according 
to a ratio of the set frame rate FRs to a frame rate when the audio 
data DAa is generated. When the frame rate FRq in generating the audio 
data DAa is n times the reference frame rate FRr and the set frame rate 
FRs is n/2 times the reference frame rate FRr, the effective data 
selection circuit 171 performs the frame-skipping on the audio data DAa 
every other sample. In this case, since an interval of the frame- 
skipping may be shortened as compared by a case where performing the 
frame-skipping on the audio data DAa on a frame basis, audio having an 
excellent sound quality may be get based on the audio data DA. 

Thus, the image data DVa having a fixed frame frequency makes 
unnecessary the alteration of operation frequencies in the imaging unit 
12 and the camera processing circuit 131 of the signal processor 17. 
This allows configurations of the imaging unit 12 and the camera 
processing circuif 131 to be made simpler. Since only the data 
extraction on a frame basis from the ijnage data DVa may generate ijiage 
data DV of the set frame rate FRs, generating the image data DV of a 
predetermined set frame rate FRs from the image data Dva is easily 
allowed. 

When the imaging apparatus is provided with video memory or an 
adder and a divider, it may generate the image data DV by adding the 
image data every the predetermined frames. This allows a variable range 
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of f rane rate in the iinaged signal Sp to be limited and the frame rate 
FRs to be successively altered. Adding the imaged signal Sp of n frames 
and dividing the signal level by n allows the signal having a frame 
rate 1/n times the imaged signal Sp to be obtained even if a frame rate 
of the imaged signal Sp is not divided by n. Further, successively 
altering the read-out frecjuency of imaged charge allows the set frame 
rate FRs to be successively altered. 

Figs. 8A through 8E and 9A through 9E are diagrams showing 
relationship between the image data DV generated by the imaging 
apparatus 10, 10a and the associated information EM. 

When the set frame rate FRs is set to the one equal to or -twice 
the reference frame rate FRr as shown in Fig. 8A, the associated 
information EM that includes the frame rate information CM-FRs 
indicating the set frame rate BKs, as shown in Fig. 8C, and the frame 
identification information CM-BN indicating the sub-frame number m, as 
shown in Fig. 8D is combined with image data DV as shown in Fig. SB (in 
Fig. 8B, frame images based on the image data DV are shown) . Fig. 8E 
illustrates a relationship between time and the frame images. The frame 
rate information CM-FRs may include a magnification of the set frame 
rate to the reference frame rate EEr in addition to the set frame 
rate FRs. The frame rate information CM-FRs is indicated by the 
magnification in Figs. 8E, 9C, and 12B. 

When the set frame rate FPs is set to the one equal to or half as 
much as the reference frame rate FRr as shown in Fig. 9A, the 
associated information CM that includes the frame rate information CM- 
EKs indicating the set frame rate FRs, as shown in Fig. 9C, and the 
frame identification information CM-BN indicating the sub-frame number 
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BN, as Shown in Fig. 9D is corbined with image data as shown in Fig. 
9B (in Fig. 9B, frame images based on the image data DV are shown) . Fig. 
9E illustrates a relationship between time and the frame images. 

Fig. 10 shows a configuration of signal-recording apparatus 20 
such as a videotape recorder. 

When the material data DUti that combines main data indicating 
image and/or audio with the associated infonration including frame rate 
information of the main data is input, encoder 211 of signal-recording 
processor 21 receives the material data UTm as recording data, ^e 
encoder 211 produces error-correcting codes and performs data-shuffling, 
multiplexing, channel coding, and the like based on operation control 
signal CIM received from a recording/reproducing control unit 24 using 
the material data rmn thus received so that the encoder 211 can 
generate recording signal SW. ^e encoder 211 then supplies a terminal 
212a of a changeover switch 212 with the generated recording signal SW. 

■me recording/reproducing control unit 24 supplies the changeover 
switch 212 with a switch control signal CTW. Operations of the 
changeover ^itch 212 are controlled based on the switdi control signal 
crw and thus, the recording signal SW is supplied to recording 
amplifier 213a or 213b through terminal 212b or 212c of the changeover 
switch 212. 

-me recording amplifier 213a receives and amplifies the recording 
signal SW and supplies signal-switching device 221a of signal 
recording/reproducing unit 22 with the amplified signal SW. The 
recording amplifier 213b receives and amplifies the recording signal SW 
and supplies signal-switching device 221b of the signal 
recording/reproducing unit 22 with the amplified signal SW. 
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When recording a signal, the signal-switching device 221a 
supplies magnetic head 222a with the aitplif ied recording signal SW 
received from the recording amplifier 213a based on a changeover 
control signal CWR received from the recording/reproducing control unit 
24. When reproducing a signal, the signal obtained by the magnetic head 
222a is supplied to reproducing amplifier 231a of the signal 
reproduction processor 23. Similarly, when recording a signal, the 
signal-switching device 221b supplies magnetic head 222b with the 
amplified recording signal SW received from the recording amplifiers 
213b based on a changeover control signal CWR received frcm the 
recording/reproducing control unit 24. When reproducing a signal, the 
signal obtained by the magnetic head 222b is supplied to reproducing 
amplifier 231b of the signal reproduction processor 23. 

Thus, supplying the magnetic heads 222a, 222b built in a rotating 
head drum (not shown) with the amplified recording signal SW allows 
image data and audio data, and the associated information to be 
recorded on magnetic tape, which is not shown. The magnetic heads 222a,. 
222b read out the signal recorded on the magnetic tape and supplies the 
reproducing amplifiers 231a, 231b, respectively, with the read-out 
signal . 

The reproducing amplifier 231a receives and amplifies a signal 
obtained by the magnetic head 222a and supplies a terminal 232a of a 
changeover switch 232 with the amplified signal. The reproducing 
amplifier 231b receives and amplifies a signal obtained by the magnetic 
head 222b and supplies a terminal 232b of the changeover switch 232 
with the amplified signal. A terminal 232c of the changeover switch 232 
is connected with a decoder 233. 
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The recording/reproducing control unit 24 sufplies the changeover 
switch 232 with a switch control signal CTR. Operations of the 
changeover switch 232 are controlled based on the switch control signal 
CTR on which the signals output from the recording anplifiers 213a, 
213b, respectively, are selected. The selected signal is supplied to 
the decoder 233 as reproduced signal SR. The decoder 233 decodes the 
reproduced signal SR and performs data separation, de-shuffling, an 
error-correcting processing, and the like based on operation control 
signal CIM received from the- recording/reproducing control unit 24. 
Thus, the decoder 233 generates the material data um that combines 
main data indicating image and/or audio with the associated information 
including frame rate information of the main data and then trananits 

the combined ones. 

The recording/reproducing control unit 24 is connected with 
interface unit 25. The recording/reproducing control unit 24 receives 
operational signal PSv from the interface unit 25 or another 
operational signal PSw from an external apparatus such as the editing 
apparatus 30. The recording/reproducing control unit 24 also generates 
the switch control signals CTW, CTR and the operation control signals 
CIM, cm controlling operations of the encoder 211 and the decoder 233, 
respectively, based on the operational signal PSv or PSw. The 
recording/reproducing control unit 24 further performs a driving 
control of a magnetic tape or a rotation head. The 

recording/reproducing control unit 24 generates a display signal PH and 
supplies display unit 26 with it, to display an operational condition of 
a video tape recorder, and various information and the like. 

Although a case where the material data Dim is recorded on the 
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magnetic tape has been shown in Fig. 10, the material data nay be 
recorded on disk-like optical or magnetic recording medium or signal- 
reproduction device using semiconductor storage element. 

The signal-recording apparatus 20 is enough if the iitage data DV 
and audio data DA are combined with the associated information EM and 
they are recorded or transmitted. Thus, the material data DTln to be 
input is not liitdted to the one having the associated information 
inserted into the data stream, header, or the like. The material data 
may be si?>plied through different signal lines. 

Combining the main data with the associated information CM allows 
editing processing and variable speed reproduction to be easily 
performed using the data generated by the imaging apparatus 10, 10a or 
the data recorded on the recording medium by the signal-recording 
apparatus 20. 

During an editing process of the data combining the main data 
with the associated information CM, a speed available for a 
reproduction may be set based on the set frame rate FFs. This speed is 
set so that a natural reproduction ijnage can be realized through a 
frame-skipping of a predetermined frame period or a frame repetition. 
If a multiple of the set frame rate EEs to the reference frame rate FRr 
is more than one, namely, FRs/FRr>l, the speed available for 
reproduction is set by searching for the divisors of this multiple 
other than one and calculating the reciprocals of the searched divisors. 
If the set frame rate El^ is 10 times the reference frame rate FRr, the 
divisors of this multiple, 10 other than one are of 2, 5, and 10 and 
thus, the reciprocals thereof are of 1/2, 1/5, and 1/10, respectively. 
Selection of one of the reciprocals of divisors allows the speed 
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available for reproduction to be set to less than the saine speed as 
nontal reproduction speed. When the speed available for reproduction 
not less than the same speed as the normal reproduction speed is set to 
positive integer multiples of the normal reproduction speed, the frame- 
skipping periods in each of the speeds available for reproduction may 

be equal to each other. 

If a multiple of the set frame rate FRs to the reference frame 
rate FE^ is less than one, namely, EKs/FRr<l, the speed available for 
reproduction is set by searching for the reciprocals of this multiple 
and calculating the divisors of the searched reciprocals other than one 
or integer multiples of the searched reciprocals. If the set frame rate 
FRs is 1/6 tiites the reference frame rate ERr, the divisors of the 
searched reciprocals other than one are of 2, 3, and 6 and the integer 

multiples of the reciprocals are of 6, 12, 18, , respectively. 

Selection of one of them allows the speed available for reproduction to 
be set. This allows frame-skipping periods or numbers of the frame 
repetitions in each of the speeds available for reproduction may be 
equal to each other. The speed available for reproduction of less than 
the same speed as the normal reproduction speed is set to the 
one/positive multiples the norriial reproduction speed. This allows 
numbers of the frame repetitions in each of the speeds available for 
reproduction to be equal to each other. Since, however, this is 
concerned with a repetition of images, a lower limit of the speed 
available for reproduction may be set to the same speed as the normal 
reproduction speed. 

Figs. IIA through IIC illustrate GUI representations each for an 
editing operation using the associated information. For example, in 
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each of the GUI representations, a iteterial administration browser 
window 401 as material administration display is provided at an upper- 
left side thereof; a story board window 402 as reproduction order 
display is provided at an lower-left side thereof; a monitor viewer 
window 403 as reproduced image display for displaying an linage before 
or after the editing process is provided at an upper-right side 
thereof; and a time line window 404 as reproduction tiitie order display 
is provided at a lower-left side thereof. An operation control bar 405 
is provided between the monitor viewer window 403 and the time line 
window 404. 

Fig. IIA illustrates the GUI representation where the 
reproduction speed is the same speed as the normal reproduction speed. 
Fig. IIB illustrates the GUI representation where the reproduction 
speed is a tenth the normal reproduction speed. Fig. IIC illustrates 
the GUI representation where the reproduction speed is twice as much as 
the normal reproduction speed. Locations and sharps of the material 
administration browser window 401, the story board window 402, the 
monitor viewer window 403, the tJjne line window 404, and the operation 
control bar 405 are respectively illustrative ones and thus, of course, 
they are not limited to the above locations and sharps. 

The material administration browser window 401 shows a list of 
items of the material data available for editing. For each item of the 
material data, a stamp view (a thumbnail view) for indicating a title, 
a length, contents of items of the stored material data is shown. 

The story board window 402 serves as a working area for a 
production of contents. Arranging the it&as of material data on the 
reproduced order allows the contents to be produced. 
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The monitor viewer window 403 represents not only an image based 
on the material data but also a location of reproduction and a variable 
speed bar for indicating a variable range of reproduction speed. 

The time line window 404 serves as a working area for a 
production of contents according to a more detailed method to allocate 
items of the material data along a time axis. 

The operation control bar 405 represents an operation key for 
reproducing the items of material data and the contents arranged on the 
time line window 404. 

VJhen reproducing the selected material data, the set frame rate 
FRs is determined on the basis of the associated information DM, and 
then, such a calculation that the reproduction speed FP multiplied by 
the set frame rate FRs equals a determined value FD is performed. 
Reproduction process conditions are determined on the determined value 
FD. When the reproduction speed is the same speed as the normal 
reproduction speed as shown in Fig. IIA and the set frame rate FRs is 
10 t lines the reference frame rate FElr, the reproduction speed FP 
multiplied by the set frame rate FRs equals the determined value FD as 
(1*10=10) . 

Fig. 12A illustrates images based on the image data IN where the 
set frame rate FRs is 10 times the reference frame rate FE^. Fig. 12B 
illustrates the frame rate information D^I-FRs for indicating the set 
frame rate FRs of each of the frame images. Fig. 12C illustrates the 
frame identification information DXI-BN for indicating the sub-frame 
numbers. Fig. 12D illustrates the absolute frame numbers AN of the 
frame images. Since the set frame rate FRs is 10 tijnes the reference 
frame rate FRr, there are 10 frame images of the set frame rate FRs 
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within one frame period of the reference frame rate FRr, thereby 
repeating the sub-frame numbers of zero through nine. 

When the determined value FD is set to 10, namely, FD=10, as 
shown in Figs. 12E through 12G, the iitage signal Svm is generated using 
every 10 frames, namely, with the image data of nine frames being 
skipped. This allows reproduced image having same speed as the normal 
reproduction speed to be represented on the monitor viewer 403 based on 
the image signal Svm. Fig. 12E illustrates the frame identification 
information CM-BN for indicating the sub-frame numbers. Fig. 12F 
illustrates the absolute frame numbers AN of the frame images. Fig. 12G 
illustrates frame images represented by the image signal Svm. 

An operation is carried out such that an indicated width of a 
frame image representation region 404a in the time line window 404 of 
Fig. IIA is enlarged along a direction indicated by an arrow A or a 
sliding cursor 404d represented by a short bold line in a console 
indication 403c for variable speeds in the monitor viewer 403 is slid 
on a direction indicated by an arrow B. In this case, a processing as 
the slow motion of reproduction is performed. For exartple, v*ien the 
speed available for reproduction is set to 1/10 times the normal 
reproduction speed, a reproduction speed indication 403b in the monitor 
viewer 403 is altered to "x 0.1" as shown in Fig. IIB. At the same tijne, 
the sliding cursor 404d is slid to a cursor position representing a 
tenth times the normal reproduction speed in the console indication 
403c. Since the long reproduction time is required, the indicated width 
of the frame image representation region 404a is also enlarged, as 

shown in Fig. IIB. 

If the speed available for reproduction is set to 1/10 times the 
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normal reproduction speed as shown in Fig. IIB, the determined value FD 
is set to one, namely, FI>=10* (1/10)=1. When the determined value FD is 
set to one, namely, FD=1, the image signal Svm is generated using every 
FD=1 frame of the image data DV, namely, without any frame being 
skipped. This allows the reproduced images having a tenth times the 
normal reproduction speed to be represented on the monitor viewer 403 
based on the image signal Svm, as shown in Fig. 12A. 

An operation is carried out such that an indicated width of the 
frame image representation region 404a in the time line window 404 of 
Fig. IIA is shortened along a direction indicated by an arrow B or a 
sliding cursor 404d in the console indication 403c is slid on a 
direction indicated by an arrow A in the monitor viewer 403. In this 
case, a processing as the quick motion of reproduction is performed. 
For exanple, when the speed available for reproduction is set to twice 
as much as the normal reproduction speed, a reproduction speed 
indication 403b in the monitor viewer 403 is altered to "x 2.0" as 
shown in Fig. IIC. At the same time, the sliding cursor 404d is slid to 
a cursor position representing twice of the normal reproduction speed 
in the console indication 403c. Since the short reproduction time is 
required, the indicated width of the frame image representation region 
404a is shortened, as shown in Fig. IIC. 

If the speed available for reproduction is set to twice of the 
nontal reproduction speed as shown in Fig. IIC, the determined value FD 
is set to 20, namely, FI>=10*2=20. When the determined value ED is set 
to 20, namely, FD=20, the image signal Svm is generated using every 20 
frames, namely, with the image data of nineteen frames being skipped. 
This allows the reproduced images having twice speed as much as the 
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normal reproduction speed to be represented on the monitor viewer 403 
based on the image signal Svm, as shown in Figs. 12H through 12 J. Fig. 
12H illustrates the frame identification information CM-BN for 
indicating the sub-frame numbers. Fig. 121 illustrates the absolute 
frame numbers AN of the frame images. Fig. 12 J illustrates frame images 
represented by the image signal Svm. 

Thus, the material data of the set frame rate FRs is altered to 
various rates using the associated information m and the altered 
material data is can±)ined with each other, thereby allowing 
predetermined contents-data for broadcasting, a distribution, and the 
like to be easily generated. Further, when the edited image data is 
ccxribined with the corresponding associated information to generate the 
contents-data, the user side performs the same processing as the one 
performed in the editing apparatus using the associated information 
included in the contents-data, thereby allowing the reproduction speed 
of contents to be altered at the user side. For example, the set frame 
rate FRs is set to the one higher than the reference frame rate FRr to 
generate the material data such as sports event casting and then, using 
the material data, the contents-data including the associated 
information is generated. This allows only a predetermined scene to be 
seen in a slow motion by the user with other scenes being seen in 
noi^ral reproduction speed. 

The invention has been described in detail with reference to 
specific eiribodiments of the invention in the form of the imaging 
apparatus 10 having a communication function. It should be understood 
that the invention is not be limited to the errilDodiinents and that the 
invention can be applied equally well to other types of electronic 
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apparatuses. While the foregoing specification has described preferred 
embodiment (s) of the present invention, one skilled in the art may make 
many modifications to the preferred embodiment without departing from 
the invention in its broader aspects- The appended claims therefore are 
intended to cover all such modifications as fall within the true scope 
and spirit of the invention. 



